Lending direction to neural networks
We present a general formulation for a network of stochastic directional
units. This formulation is an extension of the Bolzmann machine in which the
units are not binary, but take on values on a cyclic range, between 0 and 2Pi
radians. This measure is appropriate to many domains representing cyclic or
angular values, e.g., wind direction, days of the week, phases of the moon.
The state of each unit in a directional-unit Boltzmann machine (DUBM) is
described by a complex variable, where the phase component specifies a
direction; the weights are also complex variables. We associate a quadratic
energy function, and corresponding probability, with each DUBM configuration.
The conditional distribution of a unit's stochastic state is a circular
version of the Gaussian probability distribution, known as the von Mises
distribution. In a mean-field approximation to a stochastic DUBM, the phase
component of a unit's state represents its mean direction, and the magnitude
component specifies the degree of certainty associated with this direction.
This combination of a value and a certainty provides additional
representational power in a unit. We prove that the settling dynamics for a
mean-field DUBM cause convergence to a free energy minimum. Finally, we
describe a learning algorithm and simulations that demonstrate a mean-field
DUBM's ability to learn interesting mappings.
Retrieve Paper (pdf)